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ABSTRACT 


The purpose of this thesis is to analyze Army weapon system Operational Tests & 
Evaluations (OT&E) that have been conducted at Fort Hunter-Liggett, California. Four weapon 
systems were reviewed and analyzed including: the ADATS (LOS-F-H) air defense system. 
Avenger (Pedestal Mounted Stinger) air defense system, OH-58D (AHIP) scout helicopter and 
the Apache (AH-64) attack helicopter. The most common issues that Program Managers 
encountered in preparing their systems for operational testing were identified. A comparative 
analysis is conducted and a summary of the issues and the reasons they occurred along with 
proposed solutions, enhancement observations, and ’lessons learned’ is provided for Program 
Managers. This thesis concludes that Program Managers need to place greater emphasis on the 
areas of test schedules, technical manuals, resources, and user training. 
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I 


ZMTRODDCTION 


A. PURPOSE 

The purpose of this thesis is to analyze Army weapon 
systems Operational Tests & Evaluations (0T6E) that have been 
conducted at Fort Hunter-Liggett, California. Many systems 
have not been fully configured or prepared for operational 
effectiveness and suitability testing under realistic field 
conditions. Even though the operational test may be viewed as 
the culmination of the weapon systems development process, 
many Program Managers have little experience in this area and 
as a result, significant problems are encountered during OT&E. 
This thesis provides an analysis of these problems and the 
reasons they occurred along with proposed solutions, 
enhancement observations, and 'lessons learned* that should be 
useful to Program Managers. 

B. BACKGROUND 

A major issue that frequently confronts Program Managers 
and operational test centers is the arrival of a system for 
testing when they are not fully ready. These readiness issues 
have resulted in delayed or truncated testing, and unscheduled 
expenditure of large amounts of money to resolve problems with 
'quick fixes'. In the current restrictive budget environment. 
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these problems can also result in a weapon system's 
cancellation. 

Many of these readiness issues may be quite simple to fix 
or prevent. The many mistakes that are reported may be due to 
the inexperience of project team members, since they may 
experience an operational test only once in several years. 
Due to rotation of project personnel, much knowledge may be 
lort and therefore systems may be subject to repeated 
problems. 

C. THESIS OBJECTIVE 

The objective of this thesis is to identify the most 
common problems and enhancements that have affected Army 
systems undergoing operational testing. This information will 
point out to members of the program Manager's Office and the 
Operational Test Centers the importance of these areas and 
encourage them to give these areas more emphasis. Ultimately, 
this information can lead to a smoother, less costly, and more 
accurate Operational Test & Evaluation process. A 'lessons 
learned' summary will then be developed into a format that can 
be useful to Program Managers as a guide or checklist prior to 
the operational test of their systems. 

D. RESEARCH QUESTIONS 

The following primary research question will be addressed 
in this study: What essential tasks should the PM complete to 
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ensure that his program is ready for Operational Test and 
Evaluation? 

Subsidiary research questions are: 

(1) What are the common problems that Program Managers 
fail to identify before operational testing? 

(2) What are the common program strategy mistakes that 
Program Managers make in preparation for operational 
testing? 

(3) What are the common logistical problems that arise in 
operational testing? 

(4) How did these failures affect the testing process? 

(5) What can Program Managers do to better prepare their 
systems for operational testing? 

E. SCOPE 

The scope of this research is limited to programs that 
we'.e tested at Fort Hunter-Ligget. These include the ADATS 
(LOS-F-H) air defense system. Avenger (Pedestal Mounted 
Stinger) air defense system, OH-58D (AHIP) scout helicopter 
and the Apache (AH-64) attack helicopter. The intent is to 
evaluate operational test observations of problems and 
enhancements, and to develop a set of 'lessons learned' and 
recommendations from them. The research will also include 
observations from experienced Fort Hunter-Liggett test 
personnel. 

F. METHODOLOGY 

Research for this thesis consisted primarily of a 
literature review of Operational and Force Development Test 
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Reports located at the Fort Hunter-Ligget testing facility. 
General Accounting Office Reports, Congressional Subconunittee 
Reports, Defense Systems Management College (DSMC) lessons 
learned, and Department of Defense regulations. 

Information was also gathered through interviews with 
members of the Fort Hunter-Liggett test center. Many of these 
personnel have experienced numerous operational tests and 
possess valuable insight and first hand experience with 
testing problems. These data will be especially useful, since 
after action reports often have shortcomings. Many times 
these reports lack detail, or tend to gloss over negative 
aspects of the event in question. 

This thesis will utilize a comparative analysis of 
significant issues discovered, it will be used to determine 
relative importance and sources of the problems. 

O. THESIS OROANIZATIOM 

The thesis is divided into five chapters. Chapter II 
provides a review of the acquisition process and applicable 
guidelines within the Department of Defense. It also provides 
a background of the test and evaluation process and its 
purpose. 

Chapter III identifies and discusses problems, 
enhancements, and 'lessons leax-ned* that have occurred with 
each weapon system during testing at Fort Hunter-Liggett. 
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Chapter IV concentrates and categorizes these observations 
and enhancements into 'lessons learned' for application to 
future programs. 

Chapter V presents conclusions and recommendations, and a 
'lessons learned' summary for utilization in future military 
operational testing. 

Many acronyms and edsbreviations are used in this thesis. 
They are defined and discussed in the attached appendices 
which provide information that should help the reader gain a 
better understanding of this area of the acquisition process. 
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II. THE ACQUISITION PROCESS 


A. BACKGROUMD 

The DoD acquisition system defines the process to be used 
to plan, design, develop, acquire, maintain, and dispose of 
all equipment, facilities, and services in DoD. The system 
has been continuously revised to streamline the acquisition 
process, provide for formal risk analysis, and to reduce or 
eliminate costly changes later in the production cycle. 

In the early 1970's, the Department of Defense test 
policies became more formalized and placed greater enphasis on 
test and evaluation (T&E) as a continuing function throughout 
the acquisition cycle. These policies stressed the use of T&E 
to reduce acquisition risk and provide early and continuing 
estiiiiates of the system's operational effectiveness and 
operational suiteUsility. In order to meet these policy 
objectives, it is necessary to fully integrate appropriate 
test activities into the overall development 
process.[Ref. 1] 

The defense system acquisition process underwent revision 
in 1987 in an attempt to make it less costly, less time 
consuming, and more responsive to the needs of the operational 
conBnunlty. As it is now structured, the defense system life 
cycle consists of the following five phases: 
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(1) Concept Exploration/Definition 

(2) Concept Demonstration/Validation 

(3) Engineering and Manufacturing Development 

(4) Full-Rate Production/Deployment 

(5) Operational Support 

These phases are separated by key decision points 
(milestones) during which a decision authority reviews a 
program and may terminate it or authorizes it to advance to 
the next stage in the cycle. T&E results and planned T&E in 
the future play an in^ortant part in this process and are 
rigorously assessed as part of the milestone review process. 

B. THE ACQUISITION PHASES 

The following paragraphs describe the five major milestone 
decision points and five phases of the acquisition process. 
They provide a basis for understanding the mcuiagement and 
progressive decision making associated with program 
maturation.[Ref. 2] 

1. Concept Eaqploratlon/Definltion Phase 

The first phase in the acquisition process is concept 
exploration/definition. It begins after Milestone 0 grants 
concept studies approval. This phase starts with an 
assessment of the current or projected U.S. military 
capcQslllty to perform assigned missions, called a Mission Area 
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Analysis (MAA). This assessment is conducted before a new 
acquisition starts. The MAA evaluates threat, friendly 
capabilities, technological opportunities, doctrine, and new 
defense interests. The primary objective is to identify 
deficiencies and determine a more effective means of 
performing assigned tasks. The MAA may result in 
recommendations to: 

1. Initiate new acquisition programs. 

2. Change U.S. and allied concepts and doctrine. 

3. Use existing military or commercial systems. 

4. Modify or inprove an existing system, or 

5. Enter into a cooperative research and development 

progrcun with one or more allied nations. 

If the MAA results in a recommendation to initiate a 
new acquisition progreun, a mission need statement (MNS) is 
submitted to the Defense Acquisition Executive (DAE) . The MNS 
is submitted with or before a Program Objectives Memorandum 
(POM) submission in which funds are requested. 

This leads to the Milestone I decision which will mark 
the start of a new acquisition program if the decision is to 
enter into the next phase. Milestone I establishes broad 
goals and thresholds in the areas of program cost, schedule, 
and operational effectiveness and suitability. These broad 
guidelines give the Program Manager (PM) flexibility to 
develop innovative and cost-effective solutions. The 
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Milestone I decision is made by the Defense Acquisition 
Executive and is documented in an Acquisition Decision 
Memorandum (ADM) . 

2. Concept Demonstration/Validation Phase 

The next stage of the process is called the concept 
demonstration/validation (Dem/Val) phase. During this phase, 
the following events take place: 

1. The feasibility of con^eting alternatives is 
demonstrated and the most capcdjle system for fulfilling the 
mission is selected. 

2. Prototype systems are fabricated to support both design 
development, and testing and evaluation to identify areas 
of risk. 

The program office updates life-cycle costs, sends 
cuinual funding input into the Planning, Programming and 
Budgeting System (PPBS), and prepares documentation during 
this phase to assist in the Milestone II decision. The Test 
and Evaluation Master Plan (TEMP) is updated and the 
Integrated Program Summary (IPS) is prepared. The IPS 
summarizes the results of the concept demonstration/validation 
phase, identifies the program alternatives, and establishes 
esqplicit goals and thresholds for program cost, schedule, and 
operational effectiveness and suitcdDility. 
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3. Engineering and Manufacturing Development Phase 

The third stage of the acquisition process is 
Engineering and Manufacturing Development (EMD). This begins 
after the Defense Acquisition Executive makes the Milestone II 
decision and documents it in an ADM granting approval to 
proceed with the EMD (previously called the full-scale 
development or PSD) phase. During this phase, the PM 
completes system development up to the point where an economic 
decision can be made whether or not to produce the system in 
quantity. Before this decision can be made in the 
affirmative, the PM must demonstrate that all technical, 
operational, and resource requirement thresholds have been met 
and that adequate resources are available to support 
production and deployment. This is done through the 
completion of developmental testing and the conduct of 
operational testing in a realistic environment with extensive 
user participation. 

In preparation for Milestone III, the IPS and the TEMP 
are updated to describe any program changes made since 
Milestone II and to propose goal and threshold revisions, if 
appropriate. The Milestone III decision is made by the 
Secretary of Defense and recorded in an ADM. It either 
terminates the project, requests further testing, or grants 
approval to proceed with the full-rate production and 
deployment phase. 
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4. Production and Deployment Phase 

The next phase is production and deployment. During 
this phase, the PM ensures that systems are produced and 
deployed according to plans. Operational testing and 
evaluation, user training, and logistical support are key 
activities during the production and deployment phase. A 
formal review is scheduled one to two years after initial 
deployment of a system to ensure that operational readiness 
and support objectives are being met. The results of this 
review are presented for consideration for the Milestone IV 
decision, which identifies actions and resources needed to 
ensure that objectives are achieved and maintained. 

5. Operational Support Phase 

The final stage is the operational support phase. 
This involves support of the system in the field, as it is 
monitored for suited)ility and readiness. It also involves 
determinations of whether major upgrades are necessary, or if 
deficiencies warrant consideration of replacement. 

C. TEST AMD EVALUATION 

The purposes of test and evaluation in a defense system's 
development and acquisition program are to determine the 
feasibility of conceptual approaches, to minimize design risk, 
to identify design alternatives, to compare and analyze 
tradeoffs, and to estimate operational effectiveness and 
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suitability. As a system undergoes design and development, 
the emphasis in testing moves gradually from development test 
and evaluation (DT&E ), which is chiefly concerned with the 
attainment of engineering design goals, to operational test 
and evaluation (OT&E), which focuses on questions of 
operational effectiveness, suitability, and supportability. 

The integration of T&E requirements has several dimensions 
which include two broad categories of testing: Government, 
and contractor. Government tests can be further categorized 
as user tests, which are broadly operational in emphasis, and 
builder tests, which focus on achievement of development 
requirements. 

Test and evaluation encompasses the interrelationships of 
all system elements, including equipment, software, 
facilities, personnel, and procedural data. Each work 
breakdown structure (WBS) element must receive appropriate 
T&E. In most cases (e.g., software) the system element may 
have unique requirements which constrain the testing approach. 

Another T&E dimension to consider is that testing spans 
the overall acquisition life cycle. It is not simply 
something that takes place when development is complete. 
Finally, as T&E requirements are identified for the operation 
and support functions, the process can also identify the 
resources and procedures necessary for the test activities 
themselves. 
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T&E policy, described in Part 8 of DOD Instruction 5000.2, 
provides guidelines for planning and conducting test and 
evaluation. It defines and describes the major categories of 
DT&E and OT&E, and provides for exceptions such as combining 
DT&E with OT&E, T&E for special acquisition programs, T&E of 
computer software, T&E of system alterations, and joint T&E 
programs. DOD Instruction 5000.2 specifies three general 
requirements: 


a. Successful accomplishment of T&E objectives will be a 
key requirement for decisions to commit significant 
additional resources to a program or to advance it from one 
acquisition phase to another. 

b. T&E shall begin as early as possible and be conducted 
throughout the system acquisition process to assess and 
reduce acquisition risks, and to estimate the operational 
effectiveness and operational suitability of the system. 

c. The dependence on subjective judgement of system 
performance will be minimized during testing. 


In summary, there is clear policy stating test and 
evaluation program requirements, with particular emphasis on 
those programs designated as major weapon systems. Test and 
evaluation is an integral part of the systems development 
process. It begins early and extends throughout the 
acquisition life cycle. The most general objectives of the 
T&E program are 1) to assess and reduce the risk to the 
program 2) verify technical specifications and contractual 
guarantees, and 3) to estimate the operational suitability and 
effectiveness of the system. 


13 






1. Devttlopoaental Test And Evaluation 

Developmental test and evaluation is conducted 
throughout the acquisition process to ensure the acquisition 
and fielding of an effective and supportable system. DT&E 
includes test and evaluation of components and subsystems at 
all work breakdown structure (WBS) levels including preplanned 
product improvement (P3I) changes, hardware/software 
integration, and related software changes, as well as 
qualification, live fire, and production acceptance testing. 
It involves the use of simulations, models, breadboards, 
brassboards, and testbeds, as well as engineering and 
manufacturing development models or prototypes of system 
coR^onents or the system itself. 

DT&E is normally planned, conducted, and monitored by 
the developing agency. DT&E is conducted to: 

a. Assist the engineering design and development process. 

b. Verify performance objectives and specifications. 

c. Demonstrate that design risks have been minimized. 

d. Evaluate the compatibility and interoperability with 

existing or planned equipment/systems. 

e. Provide an assurance that the system/equipment is ready 

for testing in the operational environment. 

DT&E is conducted during the Concept E3q>loration/ 
Definition (C/E) phase to assist in selecting preferred 
alternative system concepts, technologies, and designs. During 
the Concept Demonstration/Validation (D/V) phase, DT&E 
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identifies and validates the preferred technical approach, 
including the identification of technical risks and feasible 
solutions. During the Engineering and Manufacturing 
Development phase, DT&E demonstrates that engineering is 
reasonably complete, that all significant design problems are 
in hand, and that the design meets its required specifications 
in all areas (such as performance, reliability, and 
maintainability) within the range of environmental pareuneters 
designed for the operational ert^loyment of the system. After 
the Production and Deployment Decision, DT&E is an integral 
part of the development, validation, and introduction of 
system changes undertaken to improve the system, to react to 
new threats, and/ or to reduce life cycle costs. 

2. Qualification Testing 

As part of DT&E, each developing agency is also 
responsible for the qualification testing that verifies the 
design and the manufacturing process and provides a baseline 
for subsequent acceptance tests. Qualification tests consist 
of pre-production and production qualification tests. 

Pre-production qualification tests are formal 
contractual tests that ensure design integrity over the 
specified operational and environmental range. These tests 
usually use pre-production or prototype hardware fabricated to 
the proposed production specifications and drawings. Such 
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tests include the reliability and maintainability 
demonstration tests required prior to production release. 

Production qualification tests are conducted for all 
production items to ensure the effectiveness of the 
manufacturing process, equipment, and procedures. All new 
production items are subjected to first article tests to 
verify specification compliance and form, fit and function. 
Production acceptance tests are also conducted on each item or 
on a sample lot taken at random from each production lot. 
These tests are repeated when the process or design is changed 
significantly, and when a second or alternative source is 
brought on line. Production qualification tests are conducted 
against the contractual requirements. 

3. Operational Test And Evaluation 

The primary purpose of operational test and evaluation 
is to verify that operationally effective and operationally 
suitable systems are approved for production that meet mission 
needs and minimum operational performance requirements of the 
operating forces.[Ref. 3] 

For major systems, OT&E is normally planned and 
conducted by a major OT&E field agency located within the DoD 
component. This Operational Test Agency (OTA) must be 
separate and Independent from the developing/procuring agency. 
The OTA is responsible for managing operational testing, 
reporting test results, and providing its independent 
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evaluation of the system being tested directly to the Military 
Service Chief or Defense Agency Director. 

The principal objectives of OT&E are to; 

a. Estimate the operational effectiveness and operational 
suitability of the system. 

b. Identify needed modifications or improvements. 

c. Provide details on tactics, doctrine, organizational, 
and personnel requirements. 

d. Provide information to uphold and verify the adequacy 
of various manuals, handbooks, supporting plans, and 
documentation. 

Modeling and simulation can assist in the T&E planning 
process and can reduce the cost of the conduct of testing. 
Areas of particular application include scenario development 
and the timing of test events; the development of objectives, 
essential elements of analysis, and measures of effectiveness; 
the identification of variables for control and measurement, 
and the development of data collection, instrumentation and 
data analysis plans. Modeling and simulation can be used to 
predict ahead of time the effects of various assumptions and 
constraints and evaluate candidate measures of effectiveness 
to help in formulation of the test design plan. 

Simulations are not a substitute for live testing 
since there are many things that cannot be adequately 
simulated by computer programs. Examples include the decision 
process and the proficiency of personnel in the performance of 


17 





their functions. Therefore, operational test and evaluation 
does not include eui operational assessment based exclusively 
on conputer modeling or simulation. It also shouldn't be 
based purely on an analysis of system requirements, 
engineering proposals, design specifications, or any other 
information that is contained in the programs 
documents.[Ref. 4] 

Although OT&E is planned and conducted by an 
independent testing activity, the Program Manager (PM) 
provides the funding. He must closely coordinate all aspects 
of test and evaluation with this organization to plan 
appropriate funding and to ensure that DT&E objectives 
coincide with OT&E objectives. 

OT&E is conducted in an environment that is as 
operationally realistic as possible. Typical operation and 
support personnel are used to obtain a valid estimate of the 
user's capability to operate and maintain the system when 
deployed under both peacetime and wartime conditions. During 
operational testing, threat representative forces should be 
used whenever possible. The items tested must sufficiently 
represent expected production models to ensure that a valid 
assessment of the system can be made. 

Normally, limited follow-on operational testing 
(POT&E) will use the same system and support equipment used in 
the operational evaluation and will test the fixes to be 
incorporated in production systems, complete deferred or 
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incomplete pre-production test and evaluation, and continue 
tactics development. FOT&E will continue until the objectives 
specified in the approved TEMP for this phase have been met, 
regardless of the date of deployment of the production 
systems. 

Other operational testing may include tests of the 
existing system in a new environment, with a new subsystem, in 
a new tactical application, or against a new threat. This 
also includes system upgrades as well as changes made to 
correct deficiencies identified during previous test and 
evaluation. [Ref. 5] Excunples of this include all 
hardware and software alterations that materially change 
system performance (operational effectiveness and suitability) 
and must therefore be adequately tested and evaluated. 

4. Combined Testing 

Since DT&E and OT&E take place during the same phases 
of the acquisition cycle, it may make sense to coordinate 
developmental and operational testing to use resources more 
efficiently in obtaining the data necessary to satisfy the 
common needs of both the developing agency and the operational 
test agency. This is called combined testing. Development 
euid operational tests can be combined with approval, when 
significant, clearly identified cost and time benefits will 
result. Of course, the test objectives of both the developing 
agency and the operational test agency will have to be 
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reflected in this combined testing situation. At the 
conclusion of testing, separate DT and OT reports must be 
submitted. 

D. THE TEST AND EVALUATION MASTER PLAN 

A major controlling document for every acquisition progreun 
is the Test and Evaluation Master Plan (TEMP), which lays out 
an overall plan for developmental and operational test and 
evaluation, designed to verify that the new equipment meets 
the requirements. 

The TEMP documents the overall structure and objectives of 
the test and evaluation program [Ref. 6] . It provides 
a frcunework within which to generate detailed test and 
evaluation plans and it documents schedule and resource 
inplications associated with the T&E program. The TEMP 
identifies the necessary DT&E and OT&E activities. It relates 
program schedule, test management strategy and structure, and 
required resources to; 

(1) Critical operational issues; 

(2) Critical technical parameters; 

(3) Minimum acceptable performance requirements; 

(4) Evaluation criteria; and 

(5) Milestone decision points. 
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The TEMP'S five part format is detailed in DoD 5000.2-M, 
as follows: 


Part I • Systaa Outllna 

• Mission Description 

e System Threat Assessment 

e Minimum Acceptable Operational Performance Requirements 
e System Description 
e Critical Technical Parameters 

Part II - Integrated Test Program Summary 
e Integrated Test Program Schedule 
e Mauiagement 

Part III • DTRB Outllna 

e DT&E Overview 
e DT&E to Date 
e Future DT&E 

e Live-Fire Test & Evaluation 

Part rv • OT&E Outline 

e OT&E Overview 
e Critical Operational Issues 
e OT&E to Date 

• Future OT&E 

Part V • Test and Evaluation Resources 
e Test Articles 

• Test Sites and Instrumentation 
e Test Support Equipment 

e Threat Systems/Simulators 
e Test Targets and E}q>endables 
e Operational Force Test Support 
e Simulators, Models and Testbeds 
e Special Requirements 
e Manpower/Personnel Training 

Figure 1 

Part I concerns system details including production and 
delivery Information and the operational and technical goals 
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euid thresholds. Part II, program summary, includes a schedule 
chart that provides an overview of the major acquisition and 
T&E events. Parts III (DT&E Outline) and IV (OT&E outline) 
describe in cjuantitative terms the scope of each major test 
period. Part V, the Test Resource Summary, identifies special 
resources required for the test program. 

The TEMP is a dynamic document that should be updated at 
milestones and whenever the program has changed significantly. 
Its contents should be factual and specific, avoiding 
generalities, and eit^hasizing quantifiable and testcUsle 
requirements, both operational and technical. Although a 
summary document, it is in^erative that pertinent, but 
integrated, facts and descriptions be included. The contents 
must describe the amount and type of system testing to be 
conducted before each milestone, and the resources 
required.[Ref. 7] 

B. TEST RESOURCES 

It is important that the necessary test resources be 
identified early in the acquisition process. These resources 
include: 

(1) Test Articles. The PM should identify the actual 
number and timing requirements for all test articles, 
including key support equipment. Specifically identify 
when prototype, engineering development, preproduction, or 
production models will be used. 


22 







(2) Z££i£__ and Instrumentation. Identify the 

specific test ranges/facilities to be used for each type of 
testing. Compare the test requirements against the 
range/facility capabilities and identify any major 
shortfalls. Also, identify instrumentation that must be 
acquired specifically to conduct the planned tests. 

(3) Test Support Equipment. Identify test support 
equipment that must be acquired specifically to conduct the 
test program. 

(4) Ibr.ga.t _Systems/Simulators. Identify the type, 

number, availcQjility, and fidelity requirements for all 
threat systems/simulators. Compare the requirements for 
all threat systems with available and projected assets and 
their capabilities. Highlight any major shortfalls. Each 
threat simulator shall be subjected to validation 
procedures to estcdslish and document a baseline comparison 
with its associated threat and to ascertain the extent of 
the operational and technical performance differences 
between the two throughout the simulator's life-cycle. 

(5) Test Targets and Expendables. Identify the type, 
number, and availability requirements for all targets, 
flares, chaff, smoke generators, kill indicators, etc. that 
will be required for each phase of testing. Identify any 
major shortfalls. 

(6) Operatio nal Force Test Support. For each test and 
evaluation phase, identify the type and timing of aircraft 
flying hours, communications station support, on-orbit 
satellite contacts/coverage, and other critical support 
required. 

(7) Simulations. Mode ls and Testbeds. For each test and 
evaluation phase, identify the system simulations required, 
including computer-driven simulation models and 
hardware/software-in-the-loop testbeds. Identify the 
resources required to validate and certify their credible 
usage or application before their use. 

(8) S pecial Requirements. Discuss requirements for any 
significant non-instrumentation capabilities and resources 
such as: special data processing/data bases, unique 
mapping/charting/geodesy products, extreme physical 
environmental conditions or restricted/special use 
air/sea/landscapes. 

(9) Test and Evaluation Fundin g Requirements. Estimate, 
by Fiscal Year and appropriation line number (progreun 
element), the funding required to pay direct costs of 
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planned testing. State, by fiscal year, the funding 
currently appearing in those lines (program elements). 
Identify any shortfalls. 

(10) Manpower/ Personnel Training. Identify manpower/ 
personnel and training requirements and limitations that 
affect test and evaluation execution. 

As system development progresses, the preliminary test 
resource requirements should be reassessed and refined and 
TEMP updated to reflect any changed system concepts. Resource 
shortfalls which Introduce significant test limitations should 
also be discussed, along with an outline of planned corrective 
action.[Ref. 8] 

The PM is responsible for developing the TEMP, including 
its content and preparation. However, since part IV concerns 
OT&E, the Operational Test Agency (OTA) is usually responsible 
for the preparation, content, and coordination of that part of 
the TEMP. Therefore, the PM must establish early liaison with 
the operational tester. This assists the OTA with complete 
integration of operational assessments and OT&E requirements 
into the TEMP.[Ref. 9] 

F. PER80MMEL AMD TRAZMIM6 

It is DoD policy that manpower, personnel, and training 
(MPT), as essential elements of integrated logistics support 
(ILS), be given explicit attention early in the acquisition 
process. Principal activities required Include determination 
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and specification of requirements based on: previous 
experience with similar systems, demographic expectations, 
design trade-offs, amd contractor incentives to meet MPT 
objectives. 


1. Personnel 

Prior to OT&E, several categories of personnel must be 
identified, trained and available when the system is ready to 
test. Billet requirements must be identified and funded for 
the following personnel categories: 


• Installation technicians to design and maintain test 
support equipment and instrumentation. 

• Operations personnel to participate in and support the 
test as 'enemy' forces, and as typical users manning the 
tested system. 

• Maintenance technicians to maintain the tested equipment 
(users), and to maintain mockups and simulators that are 
used in support of the test. 

• Supervisors to direct, oversee, support and validate the 
tests. 

• Transportation personnel to test the transportability of 
the system by land, sea and air. 


Each of these categories of personnel may require: 


• Training personnel/instructors to conduct support and user 
training prior to OT&E. 

• Training programs of instruction which must be developed 
for instructor use; and 
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• Security clearances to allow necessary personnel access to 
sensitive or classified data which are used in the system 
itself, or gathered as a result of the testing. 

The PM must identify the personnel requirements and 
skill levels necessary for the system/equipment under all 
normal conditions of readiness. Operational manning and 
maintenance manpower requirements at all applicable 
maintenance levels must be addressed. Manning levels and 
schedules should be Identified by maintenance level for each 
euiticipated field testing site. It is necessary to identify 
all training courses required for installation, operation and 
maintenance personnel, together with locations and duration of 
each course. Special training devices required for such 
courses at each location should be identified and procured. 

2. Training 

Training and training support includes the processes, 
procedures, techniques, training devices and equipment used to 
train civilian, active duty and reserve military personnel to 
operate and support a material system. This includes: 
individual euid crew training; new equipment training; initial, 
formal, and on-the-job training; and logistic support planning 
for training equipment and training device acquisitions and 
installations. [Ref. 10] 
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O. TBCHNZCAL MANUALS 


Technical manuals (TMs) and publications are an integral 
part of the system /equipment support requirements. They are 
the prime means of communicating maintenance and operation 
information to the user. Manual requirements must be planned, 
progressively monitored and updated to ensure timely 
completion and delivery for adequate logistics support. Since 
the quality of TMs affects equipment maintainability, 
personnel efficiency, safety and readiness, quality in TMs 
must be a planned objective. [Ref. 11] 

Prior to entering into a formal arrangement for a 
contractor to produce TMs, the Government is responsible for 
furnishing guidance to the contractor for the development of 
a Technical Manual Plan (TMP). The contractor has the 
responsibility of justifying and validating each manual 
recommended. Additional contractor responsibilities include 
providing engineering design, maintainability, and maintenance 
analysis documentation and assistance. 

The contractor must develop and implement a validation 
plan that the Government formally approves. The validation of 
technical manuals is a continuing effort that the Government 
is required to verify. This verification refers to the 
adequacy and accuracy of the manuals. 
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ZZZ. FORT HnNTER-LZGGETT OBSERVATZOM8 

A. FORT HUMTER-LZGOET EZFERZMEMTATZOH CENTER 

The TEXCOM Experimentation Center (TEC) was established at 
Fort Hunter-Liggett in 1956. Since that time, it has existed 
in various sizes and configurations, but always with the same 
general mission of testing new Army weapon systems. The 
TEXCOM Experimentation Center's specific mission is to: 

a. Plan, conduct, and report on Army user tests and 
experiments of doctrine, training, organization, and material. 

b. Provide advice, assistance, and guidance on test and 
experimentation to combat developers, training developers, 
material developers, system managers, material producers, 
other military services, and private industry. 

c. Conduct other tests and experiments as directed by 
Commander, TEXCOM. 

d. Provide high resolution data, other scientifically 
derived data, and analysis of these data for training, 
doctrine, organization, and material development decisions. 

e. Maintain a highly responsive, trained, combat ready 
reinforced tank company (M>1 Abrams and M-2 Bradley) for 
experimentation and testing support. 

f. Within TEXCOM, develop instriunentation programs, test 
and experimentation methodologies and design, develop. 
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procure, and maintain the instrumentation necessary to 
generate required data from field experiments. 

The TEC facility is coit^osed of instrumented range 
facilities that permit extensive and detailed computer data 
collection in support of its mission. TEC is a self reliant 
organization that is fully capable of supporting most testing 
requirements. This includes the on site design and 
manufacturing of training aids and devices which are necessary 
for successful testing. To facilitate future operational 
tests, TEC is developing a portable range instrumentation 
system that will permit testing to be conducted at locations 
other than Fort Hunter-Ligget. 


B. THE APACHE HELICOPTER 
1. Background 

The earliest system test examined by this thesis is 
the Apache Advanced Attack Helicopter (AAH) OT II, which 
occurred from June to August 1981. Operational testing for 
the Apache, designated the AH-64 Attack Helicopter, was a 
Follow-on Operational Test & Evaluation (FOT&E). The initial 
operational test (lOT/OT I) was conducted in 1976. It had 
compared two candidate systems with their respective 
baselines, each an AH-IS Cobra, and led to selection of the 
AH-64 for further development. 
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The Apache OT II was a comparative baseline, three- 
phase test conducted at the Test and Evaluation Center (TEC). 
A typical Army attack helicopter unit provided personnel and 
resources for both an AH-64 test section and the AH-IS 
baseline section. The test section consisted of three AH-64s 
and two Airborne Target and Fire Control Systems equipped AH- 
ISs to act as scouts for the AH-64s. 

The baseline section consisted of three AH-lSs and two 
OH-58 scouts. The AH-IS and AH-64 aircraft were flown in the 
same operational and threat environment. 

The three phases of the test included a training 
phase, a non-live fire phase, and a live fire phase. Force- 
on-force and one-on-many engagements, with real time casualty 
assessment, were conducted during the non-live fire phase. 
The live fire phase included firing of all AAH weapons. In 
total, over 400 flight hours were accomplished. 

The purpose of the test was to assess the military 
effectiveness of the AH-64 against the baseline aircraft it 
was to replace. The OT-II operational test report stated: 
"the performance of the AH-64 was adequate for combat, 
superior to the present attack helicopters, night capable, and 
survivable.'* There were no operational issues which were 
considered to preclude the acquisition and development of the 
system. 

DT II was conducted after OT II in June, 1982. The 
purpose was to fix performance problems and operational 
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failures which had occurred during the operational test. This 
primarily involved the Target Acquisition Designation Sight 
(TADS) which had failed to perform, thus affecting testing of 
the aircraft weapon systems. 

2. Observations 

Since there were many new, highly sophisticated 
systems from eight different PM offices integrated on the AH- 
64, a separate development test training detachment was formed 
to prepare for OT II. Preliminary flight training was 
performed using modified Cobra (AH-1) surrogate aircraft. 
Eventually, fifteen prototype training devices were developed. 
Thirteen were for the support of maintenance training, and two 
were for support of pilot/gunner training. 

To support OT II, the Apache PM established a field 
office at the test area. Included in this office were PMO 
personnel from the logistics, test and evaluation, and 
technical divisions. Although controlled by test personnel, 
these PM representatives were able to improve test continuity 
and facilitate the flow of spares and repair parts. 
Controlling spares and parts had the additional benefit of 
helping to keep PMO personnel informed of what was going on. 

Some contractor personnel were permitted at the OT II 
test site to represent the depot level of maintenance support. 
These personnel included technical writers whose job had been 
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to write the AH-64 Technical Manuals (TMs). During the 
operational tests, the technical writers were able to evaluate 
recommended TM changes from crew members and support 
personnel, and update publications on the spot. These updates 
were then passed back to the users during the test. 

During the operational test, the Target Acquisition 
Designation System (TADS) was undergoing developmental and 
operational testing at the same time. The development 
schedule hadn't allowed enough time for qualification testing 
(a DT&E activity) of the TADS prototype prior to a full field 
test of the total aircraft system. There also hadn't been 
time to introduce changes to correct TADS problems discovered 
in early developmental tests. As a result, the TADS performed 
poorly and was unreliable during the operational test. 

Problems with the weapon systems were experienced from 
the start of the weapons subtests, and only one trial was 
successfully completed with no aborts or difficulties. 
Weapons control failures, aircraft generator failures, random 
pylon articulation, and dirt inside the 30mm gun's receiver 
group were all problems that curtailed testing. 

These failures Included the 2.75" Folding Fin Aerial 
Rocket (FFAR) engagements which experienced widely dispersed 
impacts over a 2 kilometer area. This was attributed to 
problems with the TADS and articulation of the weapons 
pylon.[Ref. 12] Also, due to the limited firing of 
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the SOmrn gun and the FFAR, only one simulated engagement with 
the new HELLFIRE missile was accon^lished. 

The i^ache tests also experienced significant test 
limitations due to ineffective threat weapons simulators. The 
SA-9 simulator, a air defense missile system, could not be 
employed quickly, and it constantly overheated. The ZSU-23-4 
simulator, a threat air defense gun, wasn't capsdale of firing 
on the move. The T-72 tank surrogates couldn't fire on the 
move either. For these reasons, they did not represent a 
realistic threat force. 

Operational testing for the Apache concluded in August 
1981. Results of the operational test showed that the system 
was superior to the Cobra attack helicopter, night capable, 
and survivable. At Milestone III, the Apache was approved for 
production. 

C. APACHE ENHANCEMENT OBSERVATIONS AND LESSONS LEARNED 
1. Test Schedules 

Lesson Learned. Test schedules should be delayed or modified 
if the system isn't ready for the operational test. 
Discussion. The TADS on the Apache hadn't completed DT&E, and 
it wasn't ready for OT&E. This had a negative affect on the 
results of all three weapons subsystem tests. The 30mm gun 
failures were partially attributed to the TADS failure, but 
that blame may have'been misplaced. As recently as Operation 
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Desert Storm, the Apache continued to experience trouble with 
the 30mm gun. [Ref. 13] This problem might not have 
occurred if the PM had delayed testing until the complete 
system was ready. 

2. Teelmioal Manuals 

Enhancement Observation. Contractor technical writers were 
included at the operational test site. 

Discussion. Having the technical writers present for OT&E 
ensured that user input was taken seriously. Problems with 
the TMs were fixed and the PM improved the quality of his 
system at the test site. 

3. Test Reports 

Lesson Learned. Test reports should include user comments and 
a sufficient discussion of problems found during the test. 
Discussion. The Apache test report lacked detail in both of 
these areas. These comments are very important in enabling 
non-participants to gain an understanding of the performance 
of the weapons system. 

4. Test Resources 
a. Tost Articles 

Lesson Learned. Operational testing should not be conducted 
on multiple new systems at the same time. 
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Discussion. The Apache was composed of multiple new systems 
integrated together. The failure of any of these systems 
affected the test data collection on the others. In this 
test, the TADS was not prepared for the test, and it had a 
negative affect upon results from the 30mm gun, the rocket 
system, and the Hellfire missile. 

Jb. Threat Systeoa/Simulator a 

Lesson Learned. Threat systems should adequately represent 
the threat forces. 

Discussion. Some limitations were reported with the threat 
systems due to their inaJDility to react quickly and fire on 
the move. This gave the Apache helicopters less incentive to 
mask or fire quickly as they would in a realistic environment. 
This could have resulted in invalidated test engagements, and 
overstated the combat effectiveness of the system 

5. User E3q>«rlence Training 

Enhancement Ob servation. Extensive user training was 
en^hasized prior to the operational test. 

Discussion. The Apache crews were formed into a separate test 
training detachment to prepare for OT&E and preliminary flight 
training «ras conducted on modified Cobra helicopters. To 
support this training, the Apache progreun developed a total of 
fifteen prototype training devices. Thirteen of these were 
dedicated to the support of maintenance training. The 


35 






operational test report did not cite any OT&E problems that 
were due to a lack of prior training. 

D. THE OH-58D HELICOPTER 

1. Background 

The Kiowa helicopter is a scout aircraft that has been 
in the Army inventory for several years. The decision to 
modify the aircraft from the OH-58C to the OH-58D in the 
Advanced Helicopter Improvement Program (AHIP) resulted in a 
major weapon system upgrade. This required Follow-on 
Operational Testing. 

Operational Testing (OT II) for the OH-58D was 
conducted at Fort Hunter-Liggett from September 1984 to 
February 1985. The test at Fort Hunter-Liggett immediately 
followed Developmental Testing which had been conducted Yuma 
Proving Ground in Arizona. 

2. Observations 

Although the Operational Test of the 0H-58D was 
considered to be successful, a number of issues were noted as 
having adversely affected its conduct.[Ref. 14] The 
first issues involved the pilot and support personnel 
individual skills Instruction which was received at pre¬ 
factory training, factory training, and Developmental Testing. 
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The Instxxiction was judged to be inadequate in the following 
four areas: 


a. Not all recjuired individual tasks were taught 

b. Combat skills were not taught 

c. Academics during pre-factory training were not 
reinforced with either flight time or training devices 

d. Not enough flight or hot cockpit time was available for 
factory training due to the fixed price contract. 

Many other factory training problems occurred. These 
included the lack of procedural trainers. The factory had one 
actual OH-58D availcdsle, and this was used for all pilot and 
support personnel training. The repeated system start-ups 
resulted in frequent system failures and caused training 
delays. All other training was conducted on chalk boards. 

Another issue was that this factory training was 
conducted from two to five months before the test. After 
training was completed the students were sent back to their 
home units where they resumed their normal duties. By the 
time of the test, many had forgotten much of what they had 
learned. 

A positive aspect of the factory training involved 
user input to improve the technical manuals. The contractor. 
Bell Helicopter, had technical writers present at the 
training. They were eUcle to incorporate changes and fix 
problems on the spot, as they were noted by mechanics and 
pilots. 
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other problems that affected testing included the 
availability of only two training areas for the first two 
weeks of crew training prior to the test. This was 
accompanied by a prohibition from using the system's lasers 
for the same time period. 

Technical restrictions also affected the test 
schedule. One of these involved a requirement for the 
issuance of Airworthiness Releases for OH-58C/D pink lights 
(Night vision Goggle search lights), which delayed some 
training. Also, the installation and verification testing of 
test instrumentation equipment on the aircraft caused further 
delays. Finally, one day prior to the start of the training 
program, the parent unit of the test personnel decided to 
establish the following additional requirements for single 
pilot NVG flight in the OH-58C: (1) radar altimeters, (2) 
blue-green level lighting, and (3) selected experienced crews. 
This created additional delays in training until the 
appropriate requirements could be met. 

Another problem area involved crew qualification to 
participate in the test. The operational test plan called for 
OH-58C crews who had gone through a Army Training Evaluation 
Program (ARTEP) with the AH-IS Cobra attack helicopter before 
the test. The crews weren't qualified as required when they 
arrived at the test site. This made additional training 
necessary. 
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In October 1984, approximately one month into testing, 
Headquarters, Department of the Army tasked Fort Hunter- 
Liggett with an additional system evaluation. This involved 
a Scout/Gun mix sensitivity experiment. The Test and 
Evaluation Center delayed testing for several days to develop 
instrumentation for this unplanned test. They subsequently 
conducted 18 trials of which only 12 were technically 
validated. Tactically, none of these additional trials were 
validated, so the test results were of limited use. 

These test delays and additional requirements had an 
affect upon the test schedule. The trials were originally 
expected to be conducted two times a day for four days a week. 
This was accelerated to two to three trials a day, seven days 
a week. This impacted on staffing support, which was too 
small for the busier schedule. The aviation support unit was 
kept extremely busy by this unexpected schedule and suffered 
from morale problems. 

Spare parts supply also had some difficulties due to 
post DT&E maintenance requirements. DT II had concluded at 
Y\juna, Arizona in August, and the maintenance that resulted had 
drawn heavily upon the spare parts available for the 
operational test. The unexpected OT II training and testing 
schedule further degraded the supply of spare parts. The 
exact affect of this shortage was not evaluated, but comments 
in the test report stated that it created a heavier workload 
for the support personnel. 
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Weapons simulator problems also occurred. TEC 
personnel believed that the Air Defense Threat Simulators, 
which were a contractor responsibility, weren't up to the test 
requirements. The simulators were supposed to replicate 
current threat air defense systems that would simulate a 
realistic operational environment for flight crews. The 
systems had technical problems as well as crew training and 
operational difficulties. This resulted in few valid 
engagements against the 0H-58DS being tested and evaluated. 

Operational testing for the OH-58D concluded in 
February 1985. At the Milestone III decision, the system was 
approved for fielding. 

E. OH>58D EMHAMCEMEMT OBSERVATIONS AND LESSONS LEARNED 
1. Test Schedules 

Lesson Learned. OT&E should not be scheduled too closely 

behind a developmental testing event. 

Discussion. The OH-58D went into its operational test one 
month behind its developmental test at Yuma. The quick 
transition didn't allow time to correct system problems or to 
fully recoup the spare parts supply from the post-DT&E 
maintenance. This caused extra work for the OT&E support 
personnel, and it could have negatively affected the 
Reliability, Availability and Maintainability (RAM) 
data collected during the test. 
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Lesson Learned. Test requirements should not be added once 
the schedule has been established, and especially after 
testing has started. 

Discussion. The last minute requirement for a Scout/Gun 
sensitivity test severely taxed the resources of Fort Hunter- 
Liggett, and resulted in worn-out crews and support personnel. 
This extra requirement resulted in the collection of data that 
were not validated and thus, were of little use. The schedule 
deviation did have a negative affect upon the test through the 
long hours and overwork that resulted. 

Lesson Learned. User training time should be planned into the 
beginning of the test schedule. 

Discussion. with this system, prior user training was 
insufficient. This resulted in a need for significant and 
unexpected retraining at Fort Hunter-Ligget prior to the test. 
This time was taken from the test schedule, and contributed to 
the overwork of the support personnel. 

2. Technical Manuals 

Enhancement Observation. Early efforts were taken to review 
and correct the TMs. 

Discussion. The contractor took advantage of the factory 
training to have the technical writers and users find and fix 
TM problems before the operational test. These problems could 
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have had an adverse affect upon the OT&E RAM data if they 
hadn't been corrected prior to the tests. 

3. Test Reports 

Enhancement Observation. A detailed test report can be of 
great value in determining problems and for developing a 
programs 'lessons learned'. 

Discussion. This thesis found the OH-58D test report to be 
the most thorough of the four systems examined. It had 
detailed user comments, and it traced test problems back to 
their sources. This resulted in a extremely informative test 
report. 

4. Test Resources 

a. Test Sites and Instrumentation 
Lesson Learned. Sufficient training areas should be made 

available at the operational test site. 

Discussion. Pre-training at Fort Hunter-Ligget was initially 
hindered by limited training areas and the inability to 
utilize the laser system on the aircraft. For the OH-58D, 
pre-training was critical to the success of the weapons 
system. 


b. Test Support Equipment 

Lesson Learned. Spare parts supplies should be available to 
support testing operations. 
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Discussion. The OH-58D spare parts supply was depleted by 
post-DT&E maintenance. This could have impacted on the OT&E 
RAM data and unnecessarily increased program risk. 

c. Threat Systems/Simulators 

Lesson Learned. Threat simulators should be adequate for the 
tests. 

Discussion. The threat systems in this test did not simulate 
an effective threat environment. This was due to mechanical 
and crew training problems on the threat systems. These 
problems could have invalidated the tests. 

d. Operational Force Test Support 

Lesson Learned. Coordinate with test support units early on, 
so that they can voice any concerns. 

Discussion. The test support unit had legitimate safety 
concerns over night flying of the weapon system, but the time 
to discover a problem is not the day before the test. If the 
support unit has a full understanding of what is going to 
occur, they can give their input and make objections early in 
the process. 

Lesson Learned. Sufficient support personnel to meet test 
requirements should be available. 

Discussion. There were insufficient support personnel 
involved in this test to handle the work load brought on by 
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the additional training. This shortfall could have negatively 
affected maintainability data and should be avoided in future 
test programs. 

5. User Baq>erlence Training 

Lesson Learned. Users should have sufficient technical and 
tactical training before the tests are scheduled to begin. 
Discussion. Even though this system was an upgrade of the OH- 
58C, significant technical and tactical differences affected 
the users. They were not qualified as required, and the 
factory training was insufficient. This resulted in extra on¬ 
site training. 

Lesson Learned. Testing should follow soon after user 

training is conducted. 

Discussion. With this system, a three to five month lag 
occurred between training and the tests. The need to conduct 
retraining used up several days of the available test time. 

Lesson Learned. Sufficient simulators should be available for 
crew and support training prior to the test. 

Discussion. Some of the most apparent problems of the OH-58D 
operational tests involved the lack of simulators or mock-ups 
for the contractor conducted factory training. This resulted 
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in extra on-site training and could have had a severe impact 
on the test results. 


F. THE AVENGER AIR DEFENSE SYSTEM 

1. Background 

The Pedestal Mounted Stinger, now known as the 
Avenger, is an air defense system consisting of Stinger 
missiles on a turret mounted on the High Mobility Medium 
Wheeled Vehicle (HMMWV). The system went through Force 
Development Testing and Experimentation (FDT&E) at Fort 
Hunter-Liggett in 1989. The Initial Operational Test & 
Evaluation followed in 1990. 

The purpose of Force Development Testing is to 
evaluate and develop tactics for the system. An advantage of 
FDT&E is that it can be used to evaluate a weapon system and 
iron out 'bugs' prior to operational testing. Another 
advantage is that it allows the users to gain valuable 
experience with the system. FDT&E results are not used for 
the Milestone III decision and give the user an opportunity 
make mistakes and learn without having the results used 
against the program. 

2. Observations 

During Avenger Force Development Testing, exploratory 
trials were conducted for li days. They had been scheduled 
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for only one week, but the sixth and final Avenger system had 
not yet arrived from the contractor. Fortunately, the testing 
schedule was flexible and the extra days were used for 
additional platoon training. It also permitted Fort Hunter- 
Liggett (FHL) to improve its data collection procedures for 
the test. 

Only two Avenger firing units were availadDle for user 
training before the Force Development Test. Three more 
arrived at FHL in time for the tests, but as noted, the sixth 
unit didn't arrive until after the scheduled test start date. 
This meant that the testing unit couldn't conduct realistic, 
full platoon training until the actual test. During previous 
training at Fort Bliss, the platoon had only been able to use 
two available systems with other surrogate vehicles. 

When the final Avenger system arrived at Fort Hunter- 
Liggett, it ceune directly from the manufacturer. Because it 
had arrived late, the new sixth system wasn't given an 
opportunity to go through an extensive mechanical shakedown or 
'burn-in' as the previous systems had done. Since this was a 
Force Development Test, RAM data wasn't recorded to determine 
if this resulted in any problems. The test report did state 
that "repair parts were not available" [Ref. 15], so 
any break-in problems would have been difficult to fix. 

Some difficulties with technical manuals were noted 
during FDT&E. One problem was that the operator manuals were 
written at a reading grade level of 11.09 years. This 
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exceeded the Army Training and Doctrine Command (TRADOC) 
standard of 8.0 years, resulting in comprehension problems for 
some operators. A similar but less severe problem occurred 
with the maintenance manuals which were found to be written at 
a grade level of 9.65 years. Both of these problems were 
subsequently corrected prior to OT&E. 

Another problem was that the Avenger launch signature 
device was found to be ineffective. Because of this, the 
firing of Stinger missiles was not discernable by attacking 
pilots or by other Avenger systems in the test platoon. This 
resulted in multiple and often unnecessary engagements by the 
weapon systems. 

A related problem was that threat aircraft did not 
have any type of kill signature. The test report stated that 
in several instances, this may have caused multiple 
engagements on one aircraft and none on others. 

The small size and compartmentalized terrain of Fort 
Hunter-Liggett limited testing scenarios and aircraft 
directions of attack to the valleys. This prevented the 
Avengers from spreading out laterally. It also enabled the 
Avenger operators to focus their attention forward down the 
length of the valley. In a more realistic operational 
scenario, the Avenger crews would be required to keep 360 
degrees of observation. 

During Avenger operations, the driver is often 
expected to dismount the vehicle as an aircraft observer while 
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the gxinner mans the system. During Force Development Testing, 
the drivers indicated a need for a method to stay in voice 
contact with the gunner while they were away from the vehicle. 
As soon as the need was identified, a 50 foot communications 
cable was provided to the driver/observer in all systems. 
This remedied a problem which probably would not have been 
identified until the operational test, when it could have 
affected system performance. 

Finally, during Force Development Testing, an ad hoc 
cell from the Air Defense Artillery School and the Air Defense 
Artillery Board was present. This cell was empowered to make 
changes in system utilization and tactics as they felt 
necessary. With members of both the school and the board 
present, many changes were approved and recorded on the spot. 
This enabled the Avenger Platoon to get immediate feedback to 
their suggestions and to quickly incorporate the new tactics. 

As a result of FDT&E, the Avenger Program Manager was 
able to incorporate many changes prior to the system's 
successful lOT&E in 1990. After the production decision was 
made, the system was fielded in time to be deployed with units 
participating in Operation Desert Shield and Operation Desert 
Storm. 
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O. AVEMOER EMHANCENENT OB8ERVETZOMS END LESSONS LEESNEO 

1. T«st Sehsdulss 

Lesson Learned. Tests should not be scheduled when the tested 
system may not be available. 

Discussion. The Avenger went to the scheduled tests even 
though all of the systems had not been delivered by the 
contractor and the crews hadn*t had full platoon training. 
This could have resulted in RAM problems and tactical 
difficulties due to a lack of training and experience, and 
should be avoided in future programs. 

2. Technical Manuals 

Enhancement Observation. Technical manuals were reviewed and 
corrected before the operational test. 

Discussion. The Force Development Tests were used to screen 
the TMs and many comprehension problems were discovered and 
subsequently corrected. If the PM hadn't gone through FDT&E, 
these problems might not have been discovered until the 
operational test, when they could have affected the RAM data. 

3. Test Reports 

Enhanceme nt Observation. Detailed test reports were written. 
Discussion. Comments from the Avenger FDT&E were used to make 
system improvements prior to the system's successful 
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operational test. Well written test reports can provide the 
PM with valuable information on his system. 


4. Test Resources 

a. Test Articles 

Lesson Learned. Sufficient test articles should be available 
for the test. 

Discussion. The Avenger only had two of the required six 
systems available for training prior to FDT&E. An the test 
site, one of the systems arrived late. This limited platoon 
training time and it could have adversely affected test 
results. 


b. Test Support Equipment 

Lesson Learned. Sufficient test support equipment should be 
available for the test. 

Discussion. The Avenger did not have any spare parts for the 
conduct of this test. Any maintenance problems due to 'burn- 
in' on the new systems would have negatively affected the 
tests. This introduced unnecessary risk into the program and 
should be avoided in future programs. 

c. Threat Systems/Simulators 

Lesson Learned. Simulators should be effective. 

Discussion. During this test, the simulators on the Avenger 
and supporting systems were not very visible. This resulted 






in unnecessary engagement difficulties which should be avoided 
in future programs. 

d. Vsar Exparience Training 

Enhancement Obser vation. Extensive training prior to OTSE 
gives the user experience and can result in feedback that 
improves the system. 

Discussion. The Avenger system had a dedicated platoon that 
conducted extensive crew training at Fort Bliss, Texas prior 
to FDT&E. This resulted in personnel who were knowledgeable 
of the system prior to the test. During the test, the users 
were able to make comments and ask for improvements, such as 
the 50 foot remote communications cable, that enhanced the 
system's effectiveness. Because this was a Force Development 
Test, the discovery of problems did not negatively influence 
the evaluation of the system and it reduced program risk. 

Enhancement Observation. Key decision makers and weapons 
experts were present at user training. 

Discussion. An advantage of this early training was that it 
permitted the presence and participation of a cell of Air 
Defense experts representing the Air Defense School and Air 
Defense Board. They were able to obtain direct user input and 
agree upon changes in utilization and tactics on the spot. 
This enhanced performance during OT&E. 
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H. THE ADAT8 AIR DEFENSE SYSTEM 


1. Baokgrotmd 

The Air Defense Anti-tank System, also known as the 
Line-of-Site Forward Heavy (1/>S-F-H), is an armored air 
defense system that was designed to operate at the forward 
edge of the battlefield. In 1986, Congress approved Army 
plans to test the system on a compressed time schedule. In 
the fall of 1987, the LOS-F-H Nondevelopmental Item Candidate 
Evaluation (NDICE) was conducted at VThite Sands Missile Range. 
Four candidates using off-the-shelf technology competed in the 
NDICE. The ADATS which had been developed by Martin-Marietta, 
was selected for further development. 

The ADATS system went through two iterations of Force 
Development Testing and Experimentation (FDT&E) in 1988 and in 

1989. The purpose of the testing was to develop and evaluate 
operator, crew, squad and platoon tactics and drills. It was 
also used to evaluate, modify as necessary, and validate the 
Test Support Package (TSP) for lOT&E. This included the 
Threat Support Package. 

lOT&E was conducted in two parts, the missile firing 
phase and the maneuver phase. The test report only describes 
the maneuver phase conducted by TEC at Fort Hunter-Liggett 
from March through May 1990. The missile firing phase was 
conducted at White Sands Missile Range, New Mexico in February 

1990. 
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2. C^servations 


The ADATS training was conducted by the prime 
contractor. The stated purpose of the training was "To train 
soldiers on specific system skills that are required to 
operate the system in lOT&E". It consisted of 200 hours of 
instruction conducted over 29 days at Fort Bliss, Texas. 

Prior to the start of the tests, the TEXCOM 
Experimentation Center conducted exploratory trials (pilot 
tests) from 27 March through 5 April. Fifteen trials, 
including one night trial, were conducted similar to the 
record trials. The objectives of the exploratory testing 
included maturation of the data collection and reduction 
procedures and resolution of instnjmentation problems. It 
also provided an opportunity for test controllers and players 
to refine their procedures. In addition, the exploratory 
testing provided data upon which to decide if procedures, 
instrumentation, and players were ready for the record 
trials.[Ref. 16j 

Fifty lOT&E trials were conducted from 9 April to 23 
May 1990. Each trial was a force-on-force battle which 
generally lasted one hour. Normally, two trials were 
conducted each day. Of the fifty trials conducted, three were 
invalidated due to computer, instrumentation or weather 
problems. 

A number of system manpower problems were discovered 
and recorded during the test. The first issue was that 
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drivers complained of poor visibility that degraded their 
ability to drive the vehicle. Unlike most armored vehicles, 
the ADATS didn't have a position for a Track Commander (TC) at 
the top of the vehicle. A vehicle TC normally has a better 
field of view and can assist the driver. The lack of a TC on 
the ADATS made maneuvering difficult for the driver whose 
visibility was partially blocked by the vehicle hull. 

Crew members also complained of a variety of physical 
discomforts inside of the system, and expressed concern over 
their immediate safety, and long term health hazards. Dust 
and exhaust seeped into the crew compartment, resulting in 
headaches, burning eyes, and lung and sinus problems. Poor 
seating and vibration also resulted in niunbness in the 
extremities and back pain. At times, these problems cause the 
crew to stop operations in order to exit and ventilate the 
vehicle. 

Cramped conditions within the vehicle caused other 
problems. The crews had to fit all of their required combat 
equipment into the vehicle, and this caused difficulty in 
moving around Inside. This problem was aggravated by the 
presence of test instrumentation which occupied the vehicle 
bustle rack where some of the field gear would have normally 
been stored. 

Another interior space issue affected the system's 
maintenance personnel. They had a difficult time replacing 
the system's power distribution assembly. The job required two 







personnel, but there was only room for one man near the 
assembly. The test report stated that the lack of work space 
and of easy access to parts resulted in maintainers working 
longer than would have otherwise been necessary. Of the 
eleven areas of Reliability, Availability and Maintainability 
(RAM) data collected, only the Mean Time To Repair at the 
organizational level met the established requirements. 

The maneuver phase of the ADATS lOT&E differed 
significantly from the FDT&E II which had been conducted at 
Fort Bliss. Some changes in doctrine, tactics, techniques, 
and procedures resulted from lessons learned in FDTE II. Other 
differences occurred because of the different focus of the 
test, and due to increased player resources which were devoted 
to the lOT&E. 

In the operational test, 14 Apache helicopters were 
provided as surrogates for Red rotary wing aircraft, compared 
to the six provided for FDTE II. This continually gave the 
required availability of six operational aircraft for test 
support. The remaining aircraft were either in maintenance or 
available for pilot training. As a result, the number of 
threat aircraft that the ADATS Platoon faced was substantially 
increased over the previous force development test. For the 
operational test. Fort Hunter-Liggett hired a team of ADATS 
systems manpower evaluators to assess the causality of 
soldier/squad level errors. For this analysis, errors were 
confined to those resulting in non-engagement of an aircraft 
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which the crew should have engaged, and errors during an 
engagement which resulted in non-intercept of the target. The 
following table categorizes the errors: 


Table 1 ERROR CAUSE CATEGORY 


Error Category 

Frequency 

% of Total 

Manpower/Personnel/Training 

255 

43.0 

Reliability & Maintainability 

226 

38.1 1 

1 Doctrine/Tactics 

80 

13.5 1 

Human Factors Engineering 

19 

3.2 

Safety/Health Hazards 

5 

.8 

Other 

8 

1.4 

TOTAL 

593 

100.0 1 


The ADATS operational testing concluded with the last 
test on 23 Nay 1990. At Milestone III, the decision was made 
not to proceed into production. The primary reason, cited by 
air defense experts, was loss of the ADATS primary mission due 
to the end of the cold war. 
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I. ADATS ENHANCEMENT OBSERVATIONS AND LESSONS LEARNED 

1. Test Schedules 

Lesson Learned. The operational testing schedule should be 
based upon a system's readiness for the test. 

Discussion. The ADATS was scheduled for testing after it had 
gone through two Force Development Tests which helped to 
ensure that it was ready for testing. To further ensure its 
readiness. Fort Hunter-Liggett scheduled time for fifteen 
exploratory trials to ensure that the procedures, 
instrumentation and players were ready to start the 
operational test. 

2. Test Reports 

Lesson Learned. Test reports should be detailed. 

Discussion. The operational test report for the ADATS 
contained the fewest comments and the least detail of the four 
reports examined by this thesis. Some operator problems were 
discussed, but not in detail. This makes it difficult for the 
Program Manager or other readers to determine problems or 
possible improvements that could be made. 

Enhancement Observations. Information was summarized in easy 
to understand tcdsles. 

Discussion. The ADATS operational test report used simple 
teUsles to summarize missile launch and error cause data. This 
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method made it much easier to understand possible problem 
areas and their sources. In other reports, the reader is 
required to search through extensive data in order to find 
this information. 

3 . Test Resources 

a. Test Sites and Inatrunentation 
Lesson Learned. Instrumentation should not affect user 
performance. 

Discussion. The ADATS crew members reported problems 
operating inside of the crew/operator compartment due to the 
instrumentation. The instrumentation had the affect of 
slowing movement and reactions inside the vehicle. 

Jb. Test Support Equipment 

Enhancement Observation. Adequate test support equipment was 
provided for the test. 

Discussion. For the operational test, sufficient Apache 
helicopters were provided to allow for continued testing 
without delays from maintenance or training problems. This 
resulted in a full threat force being available for all tests. 

4. User Experience Training 

Enhancement Observation. The user platoon was well trained. 
Discussion. The ADATS had a dedicated operational test 
platoon for over a year prior to the test. This enabled the 
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crew to receive extensive training in preparation for the 
test. The ADATS platoon had 200 hours of contractor training 
and two Force Development Tests at Fort Bliss, Texas and Fort 
Hunter-Liggett prior to OT&E. This resulted in the users 
being well prepared for the test. 

Enhancement Observation. User training was used to validate 
test support requirements. 

Discussion. The ADATS user training was used to validate the 
test support package for the tests. This resulted in 
logistics support that was able to fully support the user and 
test requirements. 
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IV 


OPERATIOMAL TESTING OBSERVATIONS 


This thesis has determined that a number of common 
observations and 'lessons learned* exist in the preparation 
for and conduct of operational tests by the Program Manager. 
These lessons Involve issues that hel;ed the PM improve his 
programs readiness for testing, as well as those areas that 
detracted from weapon system's readiness. A consolidation and 
categorization of these common observations and 'lessons 
learned' is presented below: 

A. TEST SCHEDULES 

Lesson Learned. Test schedules should have some flexibility 
to allow for delays caused by training, equipment, 
instrumentation and weather problems. 

Discussion. Two out of the four tests reviewed needed 
additional time for activities other then testing at the 
beginning of the testing schedule. In one case, the time was 
gained by cramming the trials into a shorter testing period. 
This resulted in morale and support problems. In another 
case, sufficient time had been built into the schedule to 
allow for a delay, and no significant problems resulted. 
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Unplanned, additional testing requirements 


should not be added to the test schedule. 

Discussion. One of the systems experienced a last minute 
addition to the testing requirements. This threw the schedule 
off by several days and resulted in information that was of 
little use. Once the test schedule is established, additional 
testing requirements should not be added without increasing 
the total test time. 

Lesson Learned. Test schedules should be established on the 
basis of system readiness, rather than strictly on milestones. 
Discussion. Three out of the four systems went to the 
operational test before they were physically ready. This 
caused problems that could have resulted in system 
cancellation due to poor test results. PMs shouldn't test 
their system if it is not ready. 

Lesson Learned. Sufficient time should be planned in the 

schedule for system maintenance and recovery after DT&E. 
Discussion. One of the four systems reviewed was adversely 
affected by recovery from DT&E. Developmental testing is 
designed to stress parts of the weapons system. If OT&E 
follows too closely, the result may be a system that is in 
need of maintenance at the start of the test. Even if the 
system is functional, there may be a shortage of spare parts 
due to DT&E recovery operations. This will seriously impact 






upon RAM data collection. In the worst case, poor RAM data 
might result in a failure of the operational test. 

B. TECHNICAL MANUALS 

Enhancement Observation. Contractor technical writers should 
be brought to the training and testing locations to correct 
TMs as problems are noted by the users. 

Discussion. Two of the four system test reports addressed TM 
problems. In both cases, these errors were caught and 
corrected before OT&E. These errors can affect system 
operation and maintenance support. Research for this thesis 
indicates that the most practical way to screen and fix TM's 
is to involve the contractors technical writers early in the 
users training and testing. This early screening will help to 
ensure accuracy and clarity. OT&E is the wrong time to 
discover that the manuals are wrong or unclear, but if they 
are, having the technical writers present will help ensure 
that the problems are fixed on the spot. 

C. TEST REPORTS 

Lesson Learned. Test reports should give a detailed report of 
what actually happened in the test. 
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Discussion, Only one of the four test reports gave sufficient 
details of what took place during the tests. Test reports 
need to tell why things happened the way they did. The 
evaluator needs to report more then just events. For tests in 
which the human factors are involved, the most neglected 
resource in finding out why things happened the way they did 
is the test participants themselves. The tester and the PM 
should not let them go without debriefing them extensively and 
recording their explanations of why things happened the way 
they did.[Ref. 17] 

D. TEST RESOURCES 

1. Test Articles 

Lesson Learned. Sufficient test articles should be produced 
and available well before the operational test is supposed to 
start. 

Discussion. In three of the cases reviewed, the lack of 
complete test articles caused truncated and ineffective 
training prior to OT&E. In one case, the users and supporters 
had minimal hands on experience with the system before the 
operational test. In another, the testing had to be delayed 
for almost a week to permit the delivery of all of the 
required vehicles for testing. In a third, the users didn't 
have a fully functioning systems due to subsystem integration 
problems. 
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Lesson Learned. Systems should go through burn-in prior to 
the operational test. 

Discussion. In one system test, the final weapon system 
literally arrived off of the truck from the factory. While 
this did not appear to have affected the test results, it did 
introduce unnecessary risk into the program. When mean time 
between failures (MTBF) is to be tested, it is better to test 
equipment that has already been through its burn-in period. 

2. Test Sites and Instrumentation 
Lesson Learned. Test sites should be adequate in size, and 
all special clearances should be obtained. 

Discussion. This is especially important when using devices 
such as laser range finders and certain kinds of 
communications equipment. These special items frequently 
require coordination with outside agencies such as the FAA or 
the Forest Service. A ten day delay was experienced by the 
OH-58D when the system was prevented from using its laser 
range finder during scheduled testing. 

Lesson Learned. Test instrumentation should not interfere 
with user operations. 

Discussion. In one of the four tests, this was a problem. 
Instrumentation can be bulky and interfere with movements of 
the users. Instrumentation should provide the needed data. 
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but not hinder the users. It should be as unobtrusive as 
possible to the test participants. 

3. Test Support Equlpnent 

Lesson Learned. All necessary support equipment should be 
available and operable. 

Discussion. In two of the four systems examined, the Program 
Manager had extra support equipment available as back-ups 
during the test. If the test involves a baseline comparison 
with another system, or is supported by another system, make 
sure that appropriate emphasis is also placed on that system's 
availability. These extra systems assured continuous support 
and gave the operators extra training systems. 

4. Threat Systeas/Simulators 

Lesson Learned. Threat systems should actually look like the 
threat systems and not the friendly system they were derived 
from. 

Discussion. This problem occurred in one of the tests 
reviewed. Poor threat systems can cause confusion among 
personnel who are trying to identify targets. This confusion 
can result in of fratricide and failures to engage enemy 
vehicles. These kinds of actions result in data that make the 
weapon system appear ineffective. 
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Lesson Learned. Threat system crew members should fully 
understand the threat tactics they are supposed to use. 
Discussion. In some tests, improper tactics resulted in 
invalidated test results. Training for test support personnel 
such as threat crew members requires emphasis and verification 
before the test begins. 

Lesson Learned. All weapons effects simulators should be 
tested and judged to be realistic and effective prior to the 
tests. 

Discussion. There was a problem with simulators in all four 
system tests. Ineffective kill lights and flash/bang devices 
result in wasted rounds and unnecessary engagements from the 
tested weapon systems. In a realistic combat environment, 
crew members rely visual clues to determine the effectiveness 
of engagements. These visual clues are just as important in 
the operational test. A lack of preparation in this area can 
result in quantatative data that make the weapon system look 
ineffective. 

5. Operational Force Test Support 

Lesson Learned. Detailed memorandums of understanding (MOUs) 
should be executed with all military units providing test 
support personnel. 
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Discussion. One of the four tests addressed this as an area 
of en^hasis due to misunderstandings with the test support 
unit. Fort Hunter-Liggett testers believe that conflict 
resolution is a normal part of their mission, but testing 
could be iirqproved if the test support units had a more 
conplete understanding of the requirements and training that 
they must corr5)lete prior to OT&E. Failure to complete this 
requisite training results in support personnel who are not 
qualified and must be retrained. This uses up valuable time 
at the beginning of the test and could possibly endanger the 
test itself. 

Lesson Learned. Support personnel should always be fully 
infomed of the latest requirements and changes. 

Discussion. The test support units need to fully understand 
the requirements of the test so that their questions and 
objections may be satisfied well in advance. The supporting 
units often have legitimate questions involving issues such as 
safety which should not come up just prior to test execution. 
These last minute conflicts can be reduced with support unit 
involvement and understanding. 

Lesson Learned. Contractor training should be observed and 
validated. 

Discussion. In one of the programs, contractor training was 
ineffective and incomplete due to a lack of training assets 
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and funding. The PM needs to be assured that the contractor 
training is up to an agreed upon standard and that it is 
consisted when these standards have been met. This training 
needs to include sufficient hands-on training time on a 
representative system, not just mock-ups. If the PM 
recognizes that contractor training is insufficient, he needs 
to plan to correct this shortfall within the testing schedule. 

Lesson Learned. OT&E should be restricted to a reasonable 
duration. 

Discussion. Many test support personnel come from other 
military facilities for the duration of the test. Overly 
lengthy tests can result in morale problems that may impact on 
test results. According to Ft. Hunter-Liggett testers, this 
is especially true when the tests encompass major family 
oriented holidays. Tests are too important to risk adverse 
results from tired or apathetic soldiers. 


B. USER EXPERIENCE TRAINING 

Lesson Learned. The Program Manager should schedule Force 
Development testing and training prior to lOT&E. 

Discussion. Two of the systems reviewed made extensive use of 
Force Development Tests before lOT&E. This testing resulted 
in inqportant user familiarization and problem identification 
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and resolution prior to the Operational Test. While the 
primary purpose of FDTStE is to verify tactics and crew drills, 
it also allows users to interact with and learn the system. 
This helps the users to find problems and make suggestions for 
system inprovements before the operational test, when those 
problems could negatively affect the systems evaluation. 

Lesson Learned. Training should not be conducted too early, 
since there may not be sufficient production representative 
systems available to support the training, and users may 
forget the training. 

Discussion. In one system, the training was conducted several 
months too early. This required extensive retraining that ate 
into the testing schedule and caused other testing 
repercussions. 

Lesson Learned. Prototypes or detailed mock-ups need to be 
available for all training conducted before OT&E. 

Discussion. Only two of the systems examined had sufficient 
prototypes to train on before the test. There should be a 
sufficient number availeible to accommodate all personnel in 

I 

training before the operational test. 
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V. C0MCLUSI0M8 AMD RBCOMNEMDATZOM8 


A. C0NCLD8I0M8 

Because testing is a major cost and schedule driver, 
adequate planning is essential long before the start of any 
testing. Test planning and continuous coordination between 
the Program Manager, the operational tester and the contractor 
is essential to the success of the weapon system during OT&E. 

DOD decisionmakers rely on the results of operational 
testing to estimate weapon system performance. But, in the 
systems this thesis reviewed, problems occurred which may have 
adversely affected the performance which the production 
decision would have been based upon. For this reason, the PM 
needs to ensure that extensive effort has been made in the 
actual preparation for OT&E. 

In the earliest tests examined for this thesis, allowances 
were made for errors and limited success was acceptable in 
OT&E. Weaknesses in the weapon systems seemed to be treated 
as something which could be fixed after the test. Today, 
these weaknesses would most likely result in cancellation of 
the program. Program managers need to place greater emphasis 
in the areas of test schedules, manuals, reports, and 
training! 


70 



B. BPECZriC CONCLUSIONS 

The following specific conclusions were derived from the 
analysis of these four programs: 

1. Test Schedules 

This thesis has concluded that pre-established 
schedules are driving the tests, not system readiness. 
Instead of testing a system when it is ready, the tendency is 
to test the system when it is scheduled. The PM's goal is to 
field an operationally effective weapon system, and this is 
not always compatible with meeting the schedule. In the four 
systems examined. Program Managers have generally not ensured 
that their systems were fully configured and ready for 
operational testing. 

2. Technical Manuals 

This thesis concludes that early attention to 
technical manuals resulted in a more accurate product and led 
to fewer logistics support problems during OT&E. TMs are an 
integral part of system/equipment support requirements. They 
are the prime means of communicating maintenance and 
operational information to the user. Since the quality of TMs 
affects equipment maintainability, personnel efficiency, 
safety and readiness, quality in TMs should always be a 
planned objective.[Ref. 18] It is imperative that all 
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system publications and manuals be completed, reviewed and 
selectively tested prior to the beginning of operational 
testing. 


3. Test Reports 

This thesis concludes that operational test reports 
lack consistency and completeness in their depth of coverage. 
This weaknesses leads to reports that do not clearly report 
what happened in the test. The reports are not as useful to 
decisionmakers or PMs as they could be. 

4. Test Resources 

This thesis concludes that the majority of the 
problems which occurred during OT&E are directly related to 
test resource issues. Part V. of the TEMP details the 
resources required, however they do not appear to get the 
attention that they warrant. The GAO stated that "Common 
weaknesses in the quality of such testing that we have 
reported include the lack of realism, independence, and test 
resources in the planning, execution, and evaluation of the 
tests.[Ref. 19] They cited twenty-seven cases where 
important test resources were limited or not available for 
testing.[Ref. 20] In spite of this apparent history 
of problems, resources still do not get the attention they 
deserve. 
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5. U««r Bspariane* Training 


The final conclusion of this thesis is that user 
experience training and testing before the operational test is 
extremely valuable to the Program Manager and his system. 
This training helps to ensure that problems are discovered 
before the test. It is also the best way for the users to 
gain realistic experience with the system before they are 
evaluated. The sooner the user is exposed to the system, the 
better things will go during OTSE. 


C. RECOMMEMDATIOMS 

To irprove the operational testing process, this thesis 
recommends that Program Managers and testers review and 
address the most comiaon issues that have affected systems that 
have already gone through testing. The Assistant Program 
Manager (APM) for testing should address the specific issues 
of schedules, technical manuals, resources and user training. 
These are the building blocks of a successful operational 
test. 

The list of 'lessons learned* detailed in this thesis are 
an important tool which can give the PM an understanding of 
potential problem areas and how to avoid or overcome them. 
Program Manager Office personnel and testers should review 
this list before evaluating system readiness for testing. In 
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addition they should keep an active list of lessons learned 
for future Program Managers and test personnel. 

To improve the operational testing process, this thesis 
makes the following specific recommendations: 


1. Test Schedules 

• Test schedules should have some flexibility to allow for 
delays caused by training, equipment, instrumentation and 
weather problems. 

• Unplanned, additional testing requirements should not be 
added to the test schedule. 

• Test schedules should be established on the basis of 
systems readiness, rather than strictly on milestones. 

• Sufficient time should be planned in the schedule for 
system maintenance and recovery after DT&E. 


2. Technical Manuals 

• Contractor technical writers should be brought to the 
training and testing locations to correct TM’s as problems 
are noted by the users. 


3. Test Reports 

• Test reports should give a detailed report of what 
actually happened in the test. 


4. Test Resources 
a. Test Articles 

• Sufficient test articles should be produced and available 
well before the operational test is supposed to start. 
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• Systems should go through burn-in prior to the operational 
test. 


b. Test Sites and Instrumentation 

• Test sites should be adequate in size, and all special 
clearances should be obtained. 

• Test instrumentation should not interfere with user 
operations. 


c. Test Support Equipment 

• All necessary support equipment should be available and 
operable. 


d. Threat Systems/Simulators 

• Threat systems should actually look like the real threat 
systems and not the friendly system they were derived 
from. 

• Threat system crew members should fully understand the 
threat tactics they are supposed to use. 

• All weapons effects simulators should be tested and judged 
to be realistic and effective prior to the tests. 


e. Operational Force Test Support 

• Detailed memorandums of understanding (MOUs) should be 
executed with all military units providing test support 
personnel. 

• Support personnel should always be fully informed of the 
latest requirements and changes. 

• Contractor training should be observed and validated. 

• OT&E should be restricted to a reasonable duration. 
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5. ns«r Bxperl«nc« Training 


• The Program Manager should schedule Force Development 
testing and training prior to lOT&E. 

• Training should not be conducted too early, since there 
may not be sufficient production representative systems 
availed)le to support the training, and users may forget 








APPENDIX At GLOSSARY OF ACRONYMS AND ABBREVIATIONS 


AAH 

AH 

ADATS 

ADM 

AHIP 

APM 

ARTEP 

ASA(RDA) 

ASARC 

DA 

DAB 

DOD 

DODI 

DOT&E 

DSMC 

DT 

DT&E 

EMD 

FDT 

FDT&E 


Advanced Attack Helicopter 

Attack Helicopter 

Air Defense Anti-Tank System 

Acquisition Decision Memorandiam 

Advanced Helicopter Improvement Program 

Assistant Progreun Manager 

Army Training Evaluation Program 

Assistant Secretary of the Army (Research, 

Development & Acquisition) 

Army Systems Acquisition Review Council 

Department of the Army 

Defense Acquisition Board 

Department of Defense 

Department of Defense Instruction 

Director of Operational Test & Evaluation 

Defense Systems Management College 

Development Test 

Developmental Test & Evaluation 

Engineering and Manufacturing Development 

Force Development Testing 

Force Development Test & Experimentation 

Folding Fin Aerial Rocket 


FFAR 







FHL 

FOT&E 

GAO 

ILS 

IOC 

lOT&E 

IPS 

LCC 

MAA 

MNS 

MOU 

MPT 

MTTR 

OH 

OPEVAL 

ORD 

OTA 

OT I 

OT&E 

PEO 

PM 

PPBS 

P3I 

R&D 

RAM 

SECFDEF 


Fort Hunter-Ligget 

Follow-on Operational Test and Evaluation 

General Accounting Office 

Integrated Logistics Support 

Initial Operational Capability 

Initial Operational Test & Evaluation 

Integrated Program Summary 

Life Cycle Cost 

Mission Area Analysis 

Mission Need Statement 

Memorandum of Understanding 

Manpower, Personnel and Training 

Mean Time To Repair 

Observation Helicopter 

Operational Evaluation 

Operational Requirements Docximent 

Operational Test Agency 

Initial Operational Test 

Operational Test & Evaluation 

Program Executive Officer 

Program Manager/Project Manager/Product Manager 
Planning, Programming and Budgeting System 
Pre-Planned, Product Improvement 
Research and Development 

Reliability, Availability, and Maintainadiility 
Secretary of Defense 
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T&E Test and Evaluation 

TADS Target Acquisition Designation Sight 

TADSS Training Aids, Devices, Simulators and Simulations 

TEC TEXCOM Experimentation Center 

TEMP Test and Evaluation Master Plan 

TM Technical Manual 

TMP Technical Manual Plan 

TSP Test Support Package 

WBS Work Breakdown Structure 
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APPENDIX Bt GLOSSARY OP DEFINITIONS 


ACQUISITION - The process consisting of planning, designing, 
producing, and distributing a weapon system/equipment. 
Acquisition in this sense includes the conceptual, validation, 
full scale development, production, and deployment/operational 
phases of the weapon systems/equipment project. For those 
weapons systems not being procured by a project manager, it 
encompasses the entire process from inception of the 
rec[uirement through the operational phase. 

ACQUISITION DECISION MEMORANDUM - A memorandum signed by the 
milestone decision authority that documents decisions made and 
the exit criteria established as the result of a milestone 
decision review or in-process review. 

ADVANCED DEVELOPMENT - Includes all projects which have moved 
into the development of hardware for tests. 

ANALYSIS-The qualitative and/or quantified evaluation of 
information requiring technical knowledge and judgement. 
AVAILABILITY - A measure of the degree to which an item is in 
em operable and committable state at the start of a mission, 
when the mission is called for at an unknown (random) time. 
BRASSBOARD - An experimental device (or group of devices) used 
to determine feasibility and to determine teclinical and 
operational data. It normally will be a model sufficiently 
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hardened to outside of environments to demonstrate the 
technical and operational properties of immediate interest. 
It may resemble the end item, but is not intended for use as 
the end item. 

BREADBOARD - An experimental device (or group of devices) used 
to determine feasibility and to develop technical data. It 
normally will be configured only for laboratory use to 
demonstrate the technical principles of immediate interest. 
It may not resemble the end item and is not intended for use 
as the projected end item. 

DEVELOPMENT TEST & EVALUATION - That test and evaluation 
conducted to assist the engineering design and development 
process and to verify attainment of technical performance 
specifications and objectives. 

EFFECTIVENESS - The performance or output received from an 
approach or a prograun. Ideally, it is a quantitative measure 
which can be used to evaluate the level of performance in 
relation to some standard, set of criteria, or end objective. 
FOLLOW-ON OPERATIONAL TEST AND EVALUATION (FOT&E) - All OT&E 
after the Production and Deployment Decision. 

INITIAL OPERATIONAL TEST AND EVALUATION (lOT&E) - All OT&E 
prior to the Production and Deployment Decision. 

INTEGRATED PROGRAM SUMMARY - A DoD Component document prepared 
and submitted to the milestone decision authority in support 
of Milestone I, II, III, and IV reviews. It provides an 
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independent assessment of a program's status and readiness to 
proceed into the next phase of the acquisition cycle. 
INTEROPERABILITY - The cUaility of systems, units, or forces to 
provide services to, and accept services from, other systems, 
units or forces, and to use the services so exchanged to 
encUsle them to operate together effectively. 

LIFE CYCLE COST - The total cost to the Government for the 
development, acquisition, operation and logistic support of a 
system or set of forces over a defined life span. 

LOGISTICS SUPPORT - The supply and maintenance of material 
essential to proper operation of a system in the force. 
LOGISTICS SUPPORTABILITY - The degree to which the planned 
logistics (including test equipment, spares and repair parts, 
technical data, support facilities, and training) and manpower 
meet system availeU^ility and wartime usage requirements. 
MAJOR SYSTEM ACQUISITION • A system acquisition program 
designated by the SECDEF to be of such importance and priority 
as to require special management and attention. 

MISSION NEED STATEMENT - A non-system specific statement of 
operational capeibility need, prepared lAW the format in DoD 
5000.2-M. 

OPERABILITY - The design characteristic of the 
system/equipment that will assure personnel feasibility and 
optimum utilization of operator personnel. 

OPERATIONAL AVAILABILITY (AO) - An index of a weapon system's 
material readiness, including system software where 
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appliccUole, in a mission environment. It is a measure of the 
probability of an items being in a condition, generally 
referred to as "up", such that is can perform its intended 
function, within acceptable limits of degradation, when called 
upon. 

OPERATIONAL EFFECTIVENESS - The capability of the system to 
perform its intended function effectively over the expected 
range of operational circumstances, in the expected 
environment, and in the face of the expected threat, including 
countermeasures. 

OPERATIONAL REQUIREMENT - The basic requirement document for 
all DoD acquisition programs requiring research and 
development effort. 

OPERATIONAL SUITABILITY - The capability of the system, when 
operated and maintained by typical military users in the 
expected numbers and of the expected experience level, to be 
reliable, maintainable, operationally available, logistically 
supportedJle when deployed, compatible, and interoperable. 
OPERATIONAL TEST AND EVALUATION - The field test under 
realistic combat conditions, of any item (or key component of) 
weapons, ec[uipment, or munitions for the purpose of 
determining the effectiveness and suitability of the weapons, 
equipment, or munitions for use in combat by typical military 
users, and the evaluation of the results of such test. 
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PREPRODUCTION PROTOTYPE - An article in final form employing 
standard parts, representative of articles to be produced 
siobseguently in a production line. 

PRODUCTION AND DEPLOYMENT DECISION - The Milestone III 
decision by which the SECDEF reaffirms the mission need, 
confirms the system as ready for production, approves the 
system for production, and authorizes the Component to deploy 
the system to the using activity. 

PROGRAM - A plan or scheme of action designed for the 
accomplishment of a definite objective which is specific as to 
the time-phasing of the work to be done and the means proposed 
for its accomplishment, particularly in quantitative terms, 
with respect to manpower, material, and facilities 
requirements. 

PROGRAM MANAGER - The individual in the DoD who manages a 
major system acquisition program. 

PROGRAM OBJECTIVES MEMORANDUM - A biennial memorandum in 
prescribed format submitted to SECDEF in April by the DoD 
components head which recommends the total resource 
requirements and prograuns within the parameters of SECDEF's 
fiscal guidance. 

RELIABILITY - The probcdaility that an item will perform its 
intended functions for a specified period of time under stated 
conditions. 

RELIABILITY, AVAILABILITY, and MAINTAINABILITY (RAM) 
Requirement imposed on acquisition systems to ensure they are 


84 




operationally ready for use when needed, will successfully 
perform assigned functions, and can be economically operated 
and maintained within the scope of logistics concepts and 
policies. 

REQUIRED OPERATIONAL CAPABILITY (ROC) - A brief statement of 
a specific operational capability which is required in the 
mid-range period. 

SURVIVABILITY - The degree to which a system is able to avoid 
or withstand a hostile environment without suffering an 
abortive impairment of its ability to accomplish its 
designated mission. 

TEST CRITERIA - Standards by which test results and outcome 
are judged. 

TEST AND EVALUATION - Process by which a system or components 
are compared against requirements and specifications through 
testing. The results are evaluated to assess progress of 
design, performance, supportability, etc. There are three 
types of T&E - Developmental (DT&E), Operational (OT&E), and 
Production Acceptance (PAT&E) - occurring during the 
Acquisition cycle. 

THREAT - The sum of the potential strength, capabilities, and 
intentions of an enemy which can limit or negate mission 
accoir^lishment or reduce force, system, or equipment 
effectiveness. 
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TRANSPORTABILITY - The inherent capability of material to be 
moved by towing, by self-propulsion, or by carrier via 
railways, highways, waterways, pipelines, ocean, and airways. 
VULNERABILITY - The characteristics of a system which causes 
it to suffer a definite degradation as a result of having been 
subjected to a certain level of effects in a man-made hostile 
environment. 
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